Agent Mode for “Serious” Developers: Autonomous Coding with VS Code

Session Date: May 21, 2025
Duration: 15 minutes
Venue: Build 2025 Conference - DEM519
Speaker: Devin Valenciano (Senior Product Manager, VS Code Team, Microsoft)
Format: Demo Session - Seattle Only
Link: [Microsoft Build 2025 Session DEM519]

VS Code Agent Mode

Executive Summary

This focused demonstration showcases VS Code’s Agent Mode capabilities through a real-world data science challenge, proving how autonomous AI coding can transform developer productivity. Devin Valenciano demonstrates Agent Mode solving a complete Kaggle housing price prediction competition from scratch, achieving top 5% performance (300th out of 6,000 submissions) in approximately 10 minutes with minimal human intervention, highlighting the transformative potential of agentic development tools.

Key Topics Covered

?? 1. Agent Mode: Beyond Traditional AI Assistance

Defining Autonomous Coding

Devin’s Core Vision: > “Agent mode’s this really cool thing that’s built into Copilot that allows you to solve bigger problems. It’s this autonomous coding assistant that can iterate on its own progress.”

Target Audience and Philosophy: > “One of the coolest things it does is that it lets APM like me pretend to be a serious developer for a little bit or a serious data scientist as we’re about to show.”

Key Capabilities:

  • Autonomous iteration - Self-directed problem-solving without constant guidance
  • Complex problem solving - Tackling multi-step, interconnected development challenges
  • Learning acceleration - Enabling non-experts to work at expert levels
  • Real-world application - Solving actual competition-grade problems

The “Serious Developer” Democratization

Breaking Down Barriers:

  • Product managers can execute technical solutions independently
  • Cross-functional collaboration - Technical and non-technical team members working at similar levels
  • Learning through doing - Agent Mode as educational tool for complex domains
  • Expertise amplification - Making advanced techniques accessible to broader audiences

?? 2. Real-World Challenge: Kaggle Housing Price Competition

The Data Science Equivalent of LeetCode

Competition Context: > “This is what’s called a Kaggle competition. It’s the data science equivalent of LeetCode. They put full competitions online for people to put submissions up and try and solve difficult problems in data science.”

Problem Specification:

  • Domain: Housing price estimation using machine learning
  • Dataset: Training data, test data, and sample submission files
  • Objective: Predict house prices based on multiple features and variables
  • Evaluation: Root Mean Square Error (RMSE) against actual prices

Starting Materials:

Competition Package:
??? Problem description and data overview
??? Training dataset (CSV format)
??? Test dataset (CSV format)
??? Sample submission file format

Real Competition Constraints

Authentic Development Environment:

  • No pre-built solutions - Starting completely from scratch
  • Competition rules - Must follow actual Kaggle submission requirements
  • Time pressure - Demonstrating rapid solution development
  • Performance metrics - Real ranking against thousands of other submissions

?? 3. Agent Mode in Action: The Development Workflow

Autonomous Problem-Solving Sequence

The Human-Like Approach: > “It does exactly what a human would do, right? It goes and reads the description file and reads the whole thing. And it’s like, OK, I understand a little bit about what this data is looking like.”

Step-by-Step Autonomous Workflow:

Agent Mode Process:
??? 1. Competition Analysis: Fetch and read competition description
??? 2. Data Exploration: Examine training, test, and sample files
??? 3. Environment Setup: Create Jupyter notebook with Python kernel
??? 4. Package Installation: Automatically select and install required libraries
??? 5. Data Analysis: Exploratory data analysis and visualization
??? 6. Data Preprocessing: Handle missing values and feature engineering
??? 7. Model Training: Train multiple machine learning models
??? 8. Model Evaluation: Compare performance across different algorithms
??? 9. Ensemble Creation: Combine models for improved accuracy
??? 10. Submission Generation: Create competition-ready submission file

Built-in Tool Integration

Seamless VS Code Integration:

  • Fetch tool - Automatic URL retrieval for competition data
  • Jupyter notebook creation - Native notebook support with kernel selection
  • File system operations - Reading and processing multiple data files
  • Python environment management - Virtual environment and package installation
  • Code execution - Automatic cell running with user permission

?? 4. Technical Implementation Deep Dive

Model Selection Strategy

Multi-Algorithm Approach:

# Agent Mode Selected Models:
Basic Models:
??? Linear Regression - Baseline statistical approach
??? Ridge Regression - Regularized linear model
??? Lasso Regression - Feature selection through regularization  
??? Elastic Net - Combined Ridge/Lasso approach

Advanced Models:
??? Random Forest - Ensemble tree-based method
??? Gradient Boosting - Sequential improvement algorithm
??? XGBoost - Optimized gradient boosting framework

Intelligent Library Selection: > “I didn’t tell it which packages to go install. It decided based on the problem set which packages would be most useful in solving the Kaggle competition.”

Automatically Installed Packages:

  • Pandas & NumPy - Data manipulation and numerical computing
  • Matplotlib & Seaborn - Data visualization and plotting
  • Scikit-learn - Machine learning algorithms and preprocessing
  • XGBoost - Advanced gradient boosting implementation

Data Analysis and Preprocessing

Exploratory Data Analysis:

  • Missing value detection - Systematic data quality assessment
  • Distribution analysis - Understanding data characteristics and skewness
  • Correlation matrix - Identifying key features correlated with house prices
  • Log transformation - Data normalization for improved model performance

Feature Engineering: > “86 lines of data cleansing that were generated almost instantaneously, which is just fantastic.”

Preprocessing Operations:

  • Missing value imputation - Strategic handling of incomplete data
  • Categorical variable encoding - Converting string variables to numerical format
  • Feature scaling - Normalizing variables for optimal model performance
  • Data validation - Ensuring quality and consistency across datasets

Advanced Ensemble Method

Sophistication Beyond Expectations: > “It actually does something even more advanced where it takes the seven models that were generated and creates what’s called an ensemble… like a combination or an average of all the different models.”

Ensemble Benefits:

  • Improved accuracy - Combining strengths of multiple models
  • Reduced overfitting - Balancing individual model weaknesses
  • Robust predictions - More reliable results across different data scenarios
  • Competition-grade approach - Professional-level machine learning technique

? 5. Performance Results and Validation

Competition Performance Metrics

Quantified Success:

Competition Results:
??? Final Score: 14,000 RMSE
??? Ranking: 300th out of 6,000 submissions
??? Percentile: Top 5% performance
??? Development Time: Approximately 10 minutes
??? Human Intervention: Minimal (single prompt + confirmations)

Development Time Analysis

Efficiency Comparison:

  • Traditional approach - Hours to days for complete data science pipeline
  • Agent Mode approach - 10 minutes for competition-ready solution
  • Learning curve - Immediate access to expert-level techniques
  • Iteration speed - Rapid experimentation and refinement capabilities

Time Savings Breakdown: > “Even if I don’t know exactly what library does which thing I can go then research. It gives me a starting point to then go learn more efficiently.”

Key Efficiency Gains:

  • Environment setup - Automatic package selection and installation
  • Boilerplate code - Generated data processing and analysis frameworks
  • Algorithm implementation - Multiple model training without manual coding
  • Best practices - Automatic application of industry-standard techniques

?? 6. VS Code Platform Advantages

Native Notebook Integration

Competitive Advantage: > “This is something that’s like really, really a competitive advantage of VS Code is our built-in native notebook support and how those notebooks can be used directly with these agentic flows.”

Platform Differentiators:

  • Seamless integration - Agent Mode works natively with Jupyter notebooks
  • Kernel management - Automatic Python environment selection and configuration
  • Cell execution - Intelligent code running with user permission controls
  • Development continuity - Single environment for all development phases

Model Selection and Flexibility

Multi-Model Support:

Available Models in VS Code Copilot:
??? Claude 3.5 Sonnet (Preferred for deep analysis)
??? Claude 3.7 Sonnet (Demonstrated model)
??? GPT-4 Turbo (Microsoft partnership)
??? GPT-4.1-O (Latest OpenAI model)
??? Gemini (Google model in preview)

Model Selection Rationale: > “I love Claude 3.7 Sonnet. It’s my favorite of all the ones listed here… I just happen to really like Claude 3.7 for this particular project because it can go deep.”

Prompt File System

Reusable Development Patterns:

  • Custom prompt storage - Save and reuse effective prompt patterns
  • Profile integration - Tie prompts to specific VS Code profiles
  • Cross-project application - Use proven prompts across multiple projects
  • Efficiency optimization - Eliminate repetitive prompt creation

?? 7. Learning and Educational Value

Democratizing Data Science Expertise

Knowledge Transfer Mechanism: > “It’s saving me a ton of time and actually probably preventing me from learning a little bit, but it’s a pretty learning tool to work alongside.”

Educational Benefits:

  • Pattern recognition - Observing expert-level problem-solving approaches
  • Technique exposure - Introduction to advanced machine learning methods
  • Best practices - Learning industry-standard development workflows
  • Rapid iteration - Quick experimentation with different approaches

Professional Development Acceleration

Skill Building Through Observation:

  • Data science methodology - Understanding complete analytical workflows
  • Feature engineering - Learning variable transformation and selection techniques
  • Model selection - Exposure to multiple algorithm approaches and trade-offs
  • Performance evaluation - Understanding metrics and validation techniques

Learning Through Doing: > “With agent mode, I can start to learn about these topics a lot faster.”

?? 8. Limitations and Considerations

Non-Deterministic Behavior

Reality of Agentic Systems: > “Unfortunately with agentic development flow, we have these little moments where it’s like, ah, shoot, I know this should work a certain way, but it’s not quite deterministic.”

Practical Challenges:

  • Unpredictable execution - Agent Mode may occasionally pause or require guidance
  • User intervention - Need for occasional prompts to continue iteration
  • Backup strategies - Importance of having fallback plans for demonstrations
  • Expectation management - Understanding limitations of autonomous systems

Ethical Considerations

Responsible Usage: > “I don’t recommend that people are submitting Kaggle submissions. This isn’t the right way to for this to be done, but it’s a really quick way to learn.”

Appropriate Applications:

  • Learning and education - Using Agent Mode for skill development
  • Prototyping and exploration - Rapid proof-of-concept development
  • Starting point creation - Generating foundation code for human refinement
  • Pattern learning - Understanding professional development approaches

Session Highlights

“Agent mode’s this really cool thing that allows you to solve bigger problems. It’s this autonomous coding assistant that can iterate on its own progress.” - Devin Valenciano

“It lets APM like me pretend to be a serious developer for a little bit or a serious data scientist as we’re about to show.” - Devin Valenciano

“It does exactly what a human would do, right? It goes and reads the description file and reads the whole thing.” - Devin Valenciano

“86 lines of data cleansing that were generated almost instantaneously, which is just fantastic.” - Devin Valenciano

“Top 5% generated in about 10 minutes time with very little serious development on my end.” - Devin Valenciano

Technical Architecture

Agent Mode Components

VS Code Agent Mode Architecture:
??? Model Selection: Claude 3.7 Sonnet (preferred for deep analysis)
??? Built-in Tools: Fetch, file operations, notebook creation
??? Execution Environment: Python virtual environment with Jupyter
??? Permission System: User approval for code execution and tool usage
??? Integration Layer: Native VS Code notebook and kernel management

Development Pipeline

Autonomous Development Workflow:
??? Problem Analysis: Competition description parsing
??? Environment Setup: Automatic package and kernel configuration  
??? Data Pipeline: Loading, exploration, and preprocessing
??? Model Training: Multi-algorithm approach with evaluation
??? Performance Optimization: Ensemble creation and tuning
??? Output Generation: Competition-ready submission files

Implementation Guidelines

Getting Started with Agent Mode

**Prerequisites:**

- VS Code with GitHub Copilot enabled
- Agent Mode feature activated in Copilot settings
- Python environment and Jupyter support installed
- Access to model selection (Claude, GPT-4, Gemini)

**Best Practices:**

- Start with clear, specific problem descriptions
- Use prompt files for complex, reusable instructions
- Monitor execution and provide continuation prompts as needed
- Verify outputs and understand generated code before execution

Optimal Use Cases

**Ideal Applications:**

- Data science exploration and analysis
- Machine learning model development
- Rapid prototyping and proof-of-concept creation
- Learning and skill development in new domains
- Complex problem solving with multiple components

**Limitations to Consider:**

- Non-deterministic execution requiring human oversight
- Need for domain knowledge to validate results
- Ethical considerations around automated competition submissions
- Dependency on internet connectivity for model access

Advanced Techniques

**Maximizing Agent Mode Effectiveness:**

- Create reusable prompt files for common problem patterns
- Use explicit context provision with relevant files and data
- Allow agent iteration while providing continuation guidance
- Combine Agent Mode with human expertise for optimal results
- Leverage built-in VS Code tools and integrations

Resources and Further Learning

VS Code Agent Mode Resources

Data Science and Machine Learning

Model Context and Integration

About the Speaker

Devin Valenciano
Senior Product Manager
VS Code Team, Microsoft
Product Manager passionate about open source, developer enablement, and AI coding. Focuses on making advanced AI capabilities accessible to developers of all skill levels. Outdoor enthusiast who enjoys mountain activities when not improving developer productivity tools.

This demonstration session showcases the transformative potential of Agent Mode in VS Code, proving that autonomous AI coding can deliver professional-grade results while dramatically accelerating learning and development speed. The real-world Kaggle competition success demonstrates how AI agents can democratize access to expert-level capabilities across different domains and skill levels.